Method and System for Calibrating a Wearable Heads-Up Display to Produce Aligned Virtual Images in an Eye Space

ABSTRACT

A wearable heads-up display has stored therein two or more distortion models for use in generating distorted source images that when projected by the wearable heads-up display into a target eye space form virtual images are aligned in the target eye space. A method and system for determining the distortion models are disclosed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/725,949, filed 31 Aug. 2018, titled “Method and System forCalibrating a Wearable Heads-Up Display to Produce Aligned VirtualImages in an Eye Space”, the content of which is incorporated herein inits entirety by reference.

FIELD

The present disclosure relates generally to display performance ofwearable heads-up displays and particularly to positioning of content inan eye space.

BACKGROUND

A scanning light projector (SLP)-based wearable heads-up display (WHUD)is a form of virtual retinal display in which a SLP draws a raster scanonto the eye of the user. The SLP projects light over a fixed areacalled the exit pupil of the display. The display may have a single exitpupil or multiple exit pupils, each of which may receive light (orimage(s)) projected by the SLP. Multiple exit pupils are used toincrease the size of the eyebox of the display beyond what is possiblewith a single exit pupil for a given size display. The eyebox is thevolume of space within which an effectively viewable image is formed bythe display. When the eye pupil is positioned inside this volume, theuser is able to see all of the content on the virtual display.Conversely, when the eye pupil is outside of this volume, the user willnot be able to see at least some of the content on the virtual display.

To produce a display UI (or content) that is seen by the eye, severalvirtual images are typically formed in the eye space by projectingseveral source images into the eye space through the exit pupil(s) ofthe display. In displays of particular interest herein, there will be Ksource images and corresponding K virtual images for each exit pupil ofthe display, where K is the number of color channels used by theprojector. If the display has N spatially separated (or separatelyaddressable) exit pupils, there will be N times K source imagesprojected from the projector space and corresponding N times K virtualimages formed in the eye space. As an example, if the projector usesred, green, and blue color channels and the display has 4 spatiallyseparated (or separately addressable) exit pupils, there will be a setof red, green, and blue source images and corresponding red, green, andblue virtual images formed in the eye space per exit pupil, for a totalof 12 virtual images formed in the eye space. Each of these 12 virtualimages will contribute to forming the display UI in the eye space.

In general, the virtual images formed in the eye space differ from eachother because the virtual images come from different color channels ofthe projector and/or because the virtual images come from different exitpupils of the display and have experienced different optical paths, andnonlinear distortion, from the projector to the eye space. The opticalsystem of the display may be designed such that the virtual imagesproduced from the different optical paths overlap in the eye space.Thus, the display UI may be made of overlapping portions of the virtualimages. In order to produce a display UI that appears as designed, e.g.,with reference to a regular, rectilinear pixel, these overlappingportions of the virtual images that form the display UI need to bealigned in the eye space. The virtual images, or overlapping portionsthereof, are considered to be aligned in the eye space if for everypoint of the display UI, all the points of the virtual images thatshould contribute to the point of the display UI are aligned in the eyespace. That is, the contributing points of the virtual images appear tocome from the same point in the eye space.

SUMMARY

A method of calibrating a wearable heads-up display (WHUD) may besummarized as including: for each virtual image to be produced by theWHUD, (a.1) determining a distortion model for use in distorting asource image corresponding to the virtual image. Determining adistortion model (a.1) may be summarized generally as including: (a.1.1)projecting a set of test patterns by a projector of the WHUD in a selectsequence, (a.1.2) capturing an image of each test pattern projected bythe projector by a camera, (a.1.3) determining a first mapping that mapspoints in a projector space to points in a camera space from thecaptured images, (a.1.4) acquiring a second mapping that maps point inthe camera space to points in the target eye space, (a.1.5) determininga third mapping that maps points from the target eye space to points inthe projector space based on the first mapping and the second mapping,and (a.1.6) storing the third mapping as the distortion model for use indistorting the source image corresponding to the virtual image.

Determining a first mapping of (a.1.3) may include decoding the capturedimages to find correspondence between projector frame buffer pixels andcamera pixels. Decoding the captured images to find correspondencebetween projector frame buffer pixels and camera pixels may includedetecting Gray codes in the captured images.

Determining a third mapping of (a.1.5) may include identifying at leastone region of interest in the target eye space and determining the thirdmapping for the at least one region of interest in the target eye space.

Determining a distortion model of (a.1) may be performed at a firsttemperature to obtain a distortion model at the first temperature and ata second temperature to obtain the distortion model at the secondtemperature.

Projecting a set of test patterns of (a.1.1) may include: for eachposition in the select sequence, generating copies of the test patternat that position and projecting the copies of the test pattern by theprojector of the WHUD, wherein each of the copies corresponds to aunique combination of exit pupil and color channel of the WHUD.Capturing an image of (a.1.2) may include capturing an image havingimage portions corresponding to the copies of the test patterns by thecamera. Determining a first mapping of (a.1.3) may include determiningthe first mapping for each unique combination of color channel and exitpupil using the image portions of the captured image corresponding tothe unique combination of color channel and exit pupil.

The method may further include (a.2) generating the set of test patternsprior to determining a distortion model of (a.1). Generating the set oftest patterns of (a.2) may include generating at least one pattern thatcarries codes. Generating at least one pattern that carries codes mayinclude generating the at least one pattern with a m-bit Gray code,where m≥1.

Determining a first mapping of (a.1.3) may include decoding the at leastone pattern that carries codes. Generating the set of test patterns of(a.2) may include generating at least one additional pattern havingfeatures with known positions in a frame buffer of the projector.Determining a first mapping of (a.1.3.) may include decoding the atleast one pattern that carries codes and the at least one additionalpattern having features with known positions in the frame buffer of theprojector.

Projecting a set of test patterns of (a.1.1) may include transmittingthe set of test patterns to a processor of the WHUD, where the processorof the WHUD renders the set of test patterns into a frame buffer of theprojector in the select sequence.

Storing the third mapping of (a.1.6) may include storing the thirdmapping in a non-transitory processor-readable storage medium of theWHUD.

Capturing an image of (a.1.2) may include positioning the camerarelative to the WHUD to capture at least a portion of images projectedby the WHUD.

A WHUD calibration system may be summarized as including: a WHUDincluding a projector, a camera positioned and oriented to captureimages projected by the projector, a calibration processorcommunicatively coupled to the WHUD and camera, and a non-transitoryprocessor-readable storage medium communicatively coupled to theprocessor. The non-transitory processor-readable storage medium storesdata and/or processor-executable instructions that, when executed by thecalibration processor, cause the calibration processor to determine adistortion model that maps points from a target eye space to points in aprojector space for each virtual image to be produced in the target eyespace by the wearable heads-up display.

The WHUD may include a processor, and the calibration processor may becommunicatively coupled to the processor of the WHUD.

A method of forming a display UI on a WHUD may be summarized asincluding: (b.1) generating N times K distorted source images, whereinN≥1 and K≥1 and N times K>1, by applying N times K different distortionmodels to K source images, wherein each of the K different distortionmodels maps points from a target eye space to points in a projectorspace corresponding to one of the K source images, wherein N is thenumber of separately addressable exit pupils of the display and K is thenumber of color channels of the display, and (b.2) projecting at leasttwo of the distorted source images from the projector space to thetarget eye space, wherein the at least two of the distorted sourceimages form at least two virtual images in the target eye space that arealigned in the target eye space. Alternatively, in projecting of (b.2),each of the N times K distorted source images may be projected from theprojector space to the target eye space, wherein the N times K distortedsource images form N times K virtual images in the target eye space thatare aligned in the target eye space.

Generating distorted source images of (b.1) may include applying the Ntimes K distortion models to the K source images when rendering the Ksource images into a projector frame buffer of the WHUD.

Generating distorted source images of (b.1) may include applying the Ntimes K different distortion models to the K source images afterrendering the K source images into a projector frame buffer of the WHUD.

Projecting distorted source images of (b.2) may include generating lightand modulating the light with the at least two of the distorted sourceimages.

The foregoing general description and the following detailed descriptionare exemplary of the invention and are intended to provide an overviewor framework for understanding the nature of the invention as it isclaimed. The accompanying drawings are included to provide furtherunderstanding of the invention and are incorporated in and constitutepart of this specification. The drawings illustrate variousimplementations or embodiments of the invention and together with thedescription serve to explain the principles and operation of theinvention.

BRIEF DESCRIPTION OF DRAWINGS

In the drawings, identical reference numbers identify similar elementsor acts. The sizes and relative positions of elements in the drawingsare not necessarily drawn to scale. For example, the shapes of variouselements and angles are not necessarily drawn to scale, and some ofthese elements are arbitrarily enlarged and positioned to improvedrawing legibility. Further, the particular shapes of the elements asdrawn are not necessarily intended to convey any information regardingthe actual shape of the particular elements and have been solelyselected for ease of recognition in the drawing.

FIG. 1 is a frontal perspective view of an example wearable heads-updisplay (WHUD).

FIG. 2 is a schematic/block diagram of a setup for calibrating a WHUDaccording to one illustrative implementation of the present disclosure.

FIG. 3 is a block diagram showing interaction of a calibration processorwith an application processor of a WHUD according to one illustrativeimplementation of the present disclosure.

FIG. 4 is a schematic diagram of a sequence of patterns generated from abinary code of 4 bits.

FIG. 5 is a schematic diagram of a sequence of patterns generated from aGray code of 4 bits.

FIG. 6 is a schematic diagram of repeating patterns.

FIG. 7A is a schematic diagram of a global pattern.

FIG. 7B is a schematic diagram of a sequence of patterns based on theglobal pattern of FIG. 7A according to one illustrative implementation.

FIG. 8A is a schematic diagram of a sequence of patterns generated froma Gray code of 2 bits.

FIG. 8B is a schematic diagram of a sequence of patterns for x and ydimensions based on a Gray code of 2 bits according to one illustrativeimplementation.

FIG. 8C is a schematic diagram of Gray code patterns having 8-pixel widestripes.

FIG. 9 is a flowchart illustrating a process of determining a distortionmodel according to one illustrative implementation of the presentdisclosure.

FIGS. 10A-1 to 10A-5 are screenshots of actual test patterns used in theprocess of FIG. 9 according to one example.

FIG. 10B-1 to 10B-5 are screenshots of images captured from projectingthe test patterns of FIGS. 10A-1 to 10A-5, respectively, according toone example.

FIG. 11 is an image of a logo or an example display UI.

FIG. 12 is a screenshot of a projector frame buffer containing distortedsource images corresponding to the image or display UI of FIG. 11according to one example.

DETAILED DESCRIPTION

In the following description, certain specific details are set forth inorder to provide a thorough understanding of various disclosedimplementations or embodiments. However, one skilled in the relevant artwill recognize that implementations or embodiments may be practicedwithout one or more of these specific details, or with other methods,components, materials, etc. In other instances, well-known structuresassociated with portable electronic devices and head-worn devices havenot been shown or described in detail to avoid unnecessarily obscuringdescriptions of the implementations or embodiments. For the sake ofcontinuity, and in the interest of conciseness, same or similarreference characters may be used for same or similar objects in multiplefigures. For the sake of brevity, the term “corresponding to” may beused to describe correspondence between features of different figures.When a feature in a first figure is described as corresponding to afeature in a second figure, the feature in the first figure is deemed tohave the characteristics of the feature in the second figure, and viceversa, unless stated otherwise.

Unless the context requires otherwise, throughout the specification andclaims which follow, the word “comprise” and variations thereof, suchas, “comprises” and “comprising” are to be construed in an open,inclusive sense, that is as “including, but not limited to.”

In the present disclosure, reference to “one implementation” or “animplementation” or to “one embodiment” or “an embodiment” means that aparticular feature, structures, or characteristics may be combined inany suitable manner in one or more implementations or one or moreembodiments.

In the present disclosure, the singular forms “a,” “an,” and “the”include plural referents unless the content clearly dictates otherwise.It should also be noted that the term “or” is generally employed in itsbroadest sense, that is, as meaning “and/or” unless the content clearlydictates otherwise.

The headings and Abstract of the present disclosure are for convenienceonly and do not interpret the scope or meaning of the embodiments orimplementations.

The following citations may be referenced in the present disclosure:

“Camera Calibration with OpenCV, Open CV 2.4.13.7 Documentation”,“docs.opencv.org/2.4/doc/tutorials/calib3d/cameracalibration/cameracalibration.html”,Web. 9 Aug. 2018.

Salvi, J. et al. 2004, “Pattern codification strategies in structuredlight systems,” Pattern Recognition, vol. 37, pp. 827-849.

The following terms may be used in the present disclosure:

User—a subject wearing a wearable heads-up display.

Display UI—visual elements that will be shown in an eye space to a user.

Exit Pupil—a fixed area where light projected by a display converges.

Virtual Image—an image that exists conceptually but is not actuallyformed on a surface. For example, a scanning laser-based wearableheads-up display forms a virtual image that appears to be floatingseveral inches in front of the user.

Eye Space—the space in which the virtual image appears from theperspective of an eye of a user, typically measured in degrees or as awidth and height at a given apparent depth.

Projector Space—the addressable projection region for the projector thatgenerates each virtual image. The source for each virtual image may haveits own projector space or may share a projector space with othervirtual images. The projector space is typically measured in pixels.

Frame buffer—a memory buffer containing at least one complete frame ofdata. The term “frame buffer image” may refer to the frame(s) of datacontained in the frame buffer. The projector of a display has anassociated frame buffer. The term “frame buffer space” and “projectorspace” may be used interchangeably. In some implementations, there maynot be a physical memory buffer which contains the entire frame of data.For example, a streaming implementation might apply all the describedcalculations “on the fly” as the values are sent to the projectormodulator.

Camera Space—the addressable space of the camera, typically measured inpixels. This is typically a dense pixel grid but may also comprise asparse set of points when alternative optical sensor topologies areemployed.

X to Y mapping—a concrete implementation mapping from X to Y (e.g.,camera space to eye space), where the mapping may be implemented inseveral different ways (e.g., a lookup table or a parameterizedpolynomial). In general, where used below an X to Y mapping can beinverted to Y to X mapping using common numerical methods, and theactual implementation may use either or both of these models.

MSB—Most significant bit, also called high-order bit. This is the bitposition in a binary number having the greatest value.

LSB—Least significant bit. This is the bit position in a binary numbergiving the units value.

In one example, a scanning light projector (SLP)-based wearable heads-updisplay (WHUD) produces virtual images in an eye space by projectingsource images from N different regions of a projector frame buffer. Theoptical system of the WHUD is designed such that the virtual imagesoverlap in the eye space. However, it is often the case that the WHUD asmanufactured does not automatically produce virtual images that arealigned in the eye space. Even after the optics in the system have beenprecisely aligned, there may still be misalignments in the virtualimages in the eye space due to the unique path and unique nonlineardistortion that produced each virtual image.

A process of calibrating a WHUD to produce virtual images that arealigned in an eye space is described herein. In general, the processincludes determining distortion models that can be applied to sourceimages such that when the source images are distorted and projected tothe eye space, the distorted source images form virtual images that arealigned in target region(s) of the eye space. The target region(s) ofthe eye space may include where the virtual images overlap in the eyespace. The process may be applied whether images are coming fromdifferent regions of the projector frame buffer or from the same regionof the projector frame buffer. The process may be applied to any WHUDthat produces at least two virtual images in the eye space where the atleast two virtual images are different in terms of optical path and/ornonlinear distortion. For example, the at least two virtual images maybe characterized by unique combinations of color channel and exit pupil.As a further example, the two virtual images may differ by color channelor by exit pupil or by both exit pupil and color channel.

FIG. 1 illustrates an example of a WHUD 100 that may be calibrated toproduce virtual images that are overlapped and aligned in an eye spaceaccording to a process described herein. WHUD 100 may have an appearanceof eyeglasses (or glasses), as shown in FIG. 1, or other near-eyedisplay form, such as goggles and the like. WHUD 100 includes a supportframe 102 that is intended to be worn on a head of a user. Support frame102 carries the devices, electronics, and software that enable WHUD 100to present a display UI in an eye space. In one example, support frame102 includes a frame front 104 carrying a pair of transparent lenses 106a, 106 b and temples 108 a, 108 b attached to opposite sides of theframe front 104. A transparent combiner 114 may be integrated in lens106 a. Many of the components of WHUD 100, such as SLP and electronicsmodule, are carried by or within temples 108 a, 108 b. Frame front 104may also carry some components of WHUD 100, such as conductors thatenable communication between components carried by or within temples 108a, 108 b and antennas.

FIG. 2 is a schematic diagram showing a setup for calibrating WHUD 100to produce virtual images that are overlapped and aligned in an eyespace according to one illustrative implementation. The setup includes,besides WHUD 100, a camera 200 that is positioned relative to WHUD 100to capture images of test patterns projected by WHUD 100. The setup mayfurther include a calibration processor 202 communicatively coupled tocamera 200 and WHUD 100. Calibration processor 202 may run programinstructions that receive captured images from camera 200 and determinemappings from the eye space to the projector space based at least inpart on the captured images. In the interest of clarity and because WHUD100 may be configured in multiple ways, only a portion of WHUD 100 isshown in FIG. 2 and described. In general, the components of WHUD 100shown in FIG. 2 pertain to presenting a display UI in the eye space.

WHUD 100 includes a SLP 112 and a transparent combiner 114. Over a scanperiod (or a total range of scan orientations), SLP 112 projects lightsignals to transparent combiner 114, and transparent combiner 114selectively redirects the light to exit pupil(s) of the display. Thelight signals may be modulated with image data from the projector framebuffer. In one implementation, an optical splitter 116 is positioned inan optical path between the SLP 112 and transparent combiner 114. In oneexample, optical splitter 116 may be a faceted optical structure such asdescribed, for example, in U.S. Pat. No. 9,989,764 (“Systems, Devices,and Methods for Eyebox Expansion in Wearable Heads-up Displays”).Optical splitter 116 receives light signals from SLP 112 and redirectsthe light signals received over each of N sub-ranges of scanorientations to a respective area of the transparent combiner 114, wherethe transparent combiner 114 redirects the light signals to respectiveexit pupil(s). In one implementation, WHUD 100 has N exit pupilscorresponding to the N sub-ranges of scan orientations, where N>1.

SLP 112 includes a light engine 118 that is operable to generate visiblelight. Light engine 118 may also generate infrared light, e.g., for eyetracking purposes. Light engine 118 may have any number and combinationof light sources to generate visible light and, optionally, infraredlight. In one example, light engine 118 includes a plurality of visiblelaser diodes (not shown separately), or other visible light sources, togenerate visible light in different wavebands. As a further example, thevisible laser diodes may include a laser diode to generate red light, alaser diode to generate green light, and a laser diode to generate bluelight. Light engine 118 may include optics to combine the output beamsof the multiple laser diodes, or multiple light sources, into a singlecombined beam 126. Light signals from light engine 118 may be modulatedwith projector frame buffer data in order to project an image to exitpupil(s) of the display.

SLP 112 further includes an optical scanner 122 that receives at least aportion of beam 126 from light engine 118 and produces a deflected beam128. Deflected beam 128 is produced for each scan orientation of theoptical scanner 122. Optical splitter 116 (or transparent combiner 114)is positioned to receive deflected beam 128 from optical scanner 122. Inone example, optical scanner 122 includes at least one scan mirror, suchas a two-dimensional scan mirror or two orthogonally-oriented mono-axismirrors that each rotate about its respective axis. The mirror(s) ofoptical scanner 122 may be microelectromechanical systems (MEMS)mirrors, piezoelectric mirrors, and the like. In other implementations,optical scanner 122 may be a mirrorless optical scanner, such as a fiberoptic scanner, or may include a combination of mirror and mirrorlessoptical scanning elements.

In one implementation, transparent combiner 114 may be a holographiccombiner having at least one hologram that is responsive to visiblelight or multiplexed holograms that are responsive to visible light. Inone example, the holographic combiner may include at least N multiplexedholograms, and each one of the at least N multiplexed holograms mayconverge light received from a respective one of the N facets of opticalsplitter 116 to a respective one of the N exit pupils of the display.Alternatively, the holographic combiner may include multiplexedholograms, each of which is responsive to light from one of the colorchannels of the projector. The holographic combiner may have other typesof holograms, such as one that is responsive to infrared light. Inanother implementation, transparent combiner 114 may be a lightguide (orwaveguide) with an input coupler to receive light into the light guidefrom scanning laser projector 112 and an output coupler to output lightfrom the light guide. Optical splitter 116 may be omitted whentransparent combiner 114 is a lightguide or waveguide.

For the calibration setup, camera 200 is positioned relative to WHUD 100to capture images projected to transparent combiner 114 by SLP 112. Theposition of camera 200 relative to WHUD 100 may be fixed during thecalibration process, in which case the field of view of camera 200should be sufficient to allow capturing of an image projected to any ofthe exit pupils, or images projected to all of the exit pupils, of thedisplay without moving the camera. Alternatively, the position of camera200 relative to WHUD 100 may be adjustable during the calibrationprocess such that camera 200 can be centered at a target exit pupil whenan image is to be captured at the target exit pupil. In one example,camera 200 may be a CCD or CMOS device with a lens and a relatively highspatial resolution. In another example, camera 200 may be a device thatis light sensitive and provides intensity information about at least apoint (or small area) of a projected image. For example, camera 200 maybe an array of photodetectors to capture a relatively small number ofpixels or may even be a single photodetector (or single pixel camera).Camera 200 may be a color camera, which may allow capturing ofmultiplexed color images, or a monochrome camera.

WHUD 100 may include an application processor (AP) 130, which is anintegrated circuit (e.g., microprocessor) that runs a range of softwareof the WHUD. In the example shown in FIG. 3, AP 130 may include aprocessor 132, GPU 134, and memory 136. Processor 132 and GPU 134 may becommunicatively coupled to memory 136, which may be a temporary storageto hold data and instructions that can be accessed quickly by processor132 and GPU 134. Storage 138 may be a more permanent storage to holddata and instructions. Each of memory 136 and storage 138 may be anon-transitory processor-readable storage medium that stores data andinstructions and may include one or more of random-access memory (RAM),read-only memory (ROM), Flash memory, solid state drive, or otherprocessor-readable storage medium. Processor 132 may be a programmedcomputer that performs computational operations. For example, processor132 may be a central processing unit (CPU), a microprocessor, acontroller, an application specific integrated circuit (ASIC), system onchip (SOC) or a field-programmable gate array (FPGA).

To form a display UI in an eye space, GPU 134 may receive source imagesfrom processor 132 and write or render the source images into theprojector frame buffer, which may be transmitted to display controller142 of display engine 144. Display controller 142 may provide the framebuffer data to light engine driver 146, e.g., laser diode driver, andscan mirror driver 148. In order to project the source images to thetransparent combiner (114 in FIG. 2), light engine driver 146 uses theframe buffer data to generate drive controls for the laser diodes (orother light sources) in the light engine 118, and scan mirror driver 148uses the frame buffer data to generate sync controls for the scanmirror(s) of the optical scanner 122. During normal operation of WHUD100, any corrections to the source images to be projected to transparentcombiner 114 may be applied when rendering the source images into theprojector frame buffer, or the display controller 142 may apply thecorrections prior to providing the frame buffer data to light enginedriver 146 and scan mirror driver 148. Several corrections, such asgeometric distortions (determined by the calibration process describedherein), color correction, and other corrections due to thermal changesin the optical system, may be applied to the source images to achieve atarget display performance. The correction model may be decomposed intomultiple stages, where each stage is applied at a separate stage in theprocess. For example, a first stage of the correction model may beapplied at GPU 134, and a second stage of the correction model may beapplied at a second processor, e.g., a SLP processor.

Calibration processor 202 is communicatively coupled to camera 200 toreceive images or light intensity data captured by camera 200 during thecalibration process. The adjective “calibration” before processor isgenerally used to distinguish this processor from other processor(s)used for normal operation of the WHUD since the calibration processorcan be decoupled from the WHUD once calibration is complete; although,conceivably, the functionality of the calibration processor may beperformed by a processor used in normal operation of the WHUD. Ingeneral, a processor that executes a calibration process as describedherein may be referred to as a calibration processor. In addition,calibration processor may be a programmed computer that performscomputational operations. For example, processor may be a centralprocessing unit (CPU), a microprocessor, a controller, an applicationspecific integrated circuit (ASIC), system on chip (SOC) or afield-programmable gate array (FPGA). Although not shown, a displayscreen may be communicatively coupled to calibration processor to allowinteraction with a calibration program running on calibration processorand/or to allow calibration processor to display calibration resultsfrom the calibration program.

Calibration processor 202 may also be communicatively coupled toapplication processor 130 for calibration purposes. In FIG. 3,calibration processor 202 is shown executing instructions of acalibration program 204. Calibration program 204 may be stored in memory206 and accessed by calibration processor 202 at run time. Calibrationprogram 204 includes decision logic 208, which when executed bycalibration processor 202, in one implementation, provides test patternsin a defined sequence to AP 130. The defined sequence may be adjustedduring the calibration process. Therefore, the term “defined” is to theextent needed for decoding of captured images of the projected testpatterns. AP 130 renders the test patterns in the defined sequence intothe projector frame buffer. Camera 200 captures images of the testpatterns as the test patterns are projected by SLP 112 (in FIG. 2) byway of transparent combiner 114 (in FIG. 2). Calibration processor 202receives camera data 210 from camera 200, and the calibration program204 uses camera data 210 to determine the eye space to projector spacemappings that will be subsequently used to generate warped sourceimages.

The calibration process includes projecting test patterns from theprojector frame buffer (or projector space) to the camera space. In oneimplementation, the test patterns are structured light projectionpatterns that when projected to a scene in a defined sequence allow eachpixel of the scene to be uniquely identified. In the setup of FIG. 2,the scene is the transparent combiner. The test patterns may begenerated using structured pattern codification techniques known in theart (see, e.g., Salvi, 2004, p. 829). In one implementation, the testpatterns enable correspondence between points in the projector space andpoints in the camera space to be found. The test patterns may be formedusing one or more of structures such as stripes, grids, and dots. Thefollowing is a description of test patterns, but the test patterns thatmay be used are not limited to the ones described below.

In one example, test patterns may be generated with binary codes. Forillustration purposes, FIG. 4 shows four patterns generated from abinary code of 4 bits, where the value goes from 0 to 15. Thus, forexample, if the patterns 0 to 3 are projected in the sequence shown, apixel corresponding to the bits at position P1 will have the value 1 1 11, which is 15. In general, a binary code of 4 bits can encode 16(=2⁴)unique stripes. To encode a frame buffer using binary code, more than 4bits would be needed. For example, if the usable area of a frame bufferhas a size of 1024×512 pixels, the x axis (1024 pixels) can be encodedwith a binary code of 10 bits (10 patterns), and the y axis (512 pixels)can be encoded with a binary code of 9 bits (9 patterns). In FIG. 4, thefirst pattern 0 (corresponding to the most significant bit (MSB)) has astripe width of 8. The second pattern 1 has a stripe width of 4. Thethird pattern 2 has a stripe width of 2. The last and fourth pattern 3(corresponding to the least significant bit (LSB)) has a stripe widthof 1. Thus, the minimum stripe width is 1. The number of transitions, oredges, in a pattern determines the edge frequency of the pattern. Forexample, first pattern 0 contains 2 edges or transitions (if the patternis wrapped around) and has the lowest pattern edge frequency, and thefourth pattern 3 contains 16 edges or transitions and has the highestfrequency. Thus, the maximum number of edges or transitions is 16. A lowminimum stripe width generally correlates to a high pattern edgefrequency. In many cases, a minimum stripe width of 1 would be too smallbecause the stripe pattern would not be easy to detect.

In another example, test patterns may be generated with Gray codes. Graycode is a coding system where two consecutive integers are representedby binary numbers differing in only one digit. For illustrationpurposes, FIG. 5 shows four patterns generated from a Gray code of 4bits, where the value goes from 0 to 15. Thus, for example, if thepatterns 0 to 3 are projected in the sequence shown, a pixelcorresponding to the bits at position P1 will have the value 1 0 0 0,which is 8. As in the binary coding case, 4 bits can encode 16(=2⁴)unique stripes, and more bits will be needed to encode each axis of aframe buffer. In FIG. 5, the first pattern 0 (MSB) has a stripe width of8. The second pattern 1 has a stripe width of 8. The third pattern 2 hasa stripe width of 4. The last and fourth pattern 3 (LSB) has a stripewidth of 2. Thus, the minimum stripe width is 2 for Gray code, comparedto 1 for binary code for the same resolution. The pattern edge frequencyof the last pattern 3 of the Gray code pattern is 8, whereas it was 16for the binary code pattern described above. Thus, Gray code patternshave a lower pattern edge frequency (or spatial frequency) compared tobinary code patterns. It is generally preferable to use the lowestfrequency patterns possible because this results in the largest and mostdistinguishable stripes. Patterns with low spatial frequency are lessvulnerable to unwanted blending between white and black sections of thepattern that can result in decoding errors. Gray codes also have theuseful property that single-bit errors generally decode with an error of±1, which can simplify error correction or simply provide more gracefulfailure modes.

Gray codes may map individual pixels or group of pixels. For example, if1024 pixels are encoded with a Gray code of 10 bits (ten patterns),there will be 1:1 correspondence between Gray code and pixels, for aminimum stripe width of 2 pixels. If 1024 pixels are encoded with a Graycode of 9 bits (nine patterns), there will be a 1:2 correspondencebetween Gray code and pixels, for a minimum stripe width of 4 pixels. If1024 pixels are encoded with a Gray code of 8 bits (eight patterns),there will be a 1:4 correspondence between Gray code and pixels, for aminimum stripe width of 8 pixels. Thus, the minimum stripe width may bemade larger by mapping Gray codes to a group of pixels, but theresolution of the measurement decreases.

In another example, test patterns may be generated with reduced imagecoding according to a strategy described herein. The strategy includesgenerating detail patterns (the patterns that contain the high frequencydata or that encode the pixels) based on Gray coding. Instead ofselecting a Gray code with a number of bits that will encode an entiredimension, a Gray code with a number of bits that will encode a fractionof the dimension is selected. The pattern that encodes a fraction of thedimension (“detail pattern” or “high-density pattern”) is then repeatedalong the dimension. This is illustrated in FIG. 6, where detail pattern250 (the structure of the pattern is not shown) is repeated along thedimension. The strategy further includes generating a global patternthat is used to identify a known point or center of the repeatingpattern. The identified known point or center will serve as a referencepoint for counting the pixels of the repeating pattern. In one example,the global pattern contains a small number of features that have knownpositions in the projector frame buffer and are easy to identifyaccurately. FIG. 7A shows an example of a global pattern including fourfeatures (e.g., dots) 310 a, 310 b, 310 c, 310 d spaced D frame bufferpixels apart. FIG. 7B shows a sequence of global patterns including apositive pattern (bp) and negative pattern (bn)—the negative pattern(bn) is an inverse of the positive pattern (bp).

When decoded, the detail patterns assign each pixel (or group of pixels)a number that is locally unique but reused globally. In one example, thedetail patterns are generated from a Gray code of 2 bits. Forillustration purposes, FIG. 8A shows two patterns generated from a Graycode of 2 bits, where the value goes from 0 to 3. The first pattern 0(MSB) has a stripe width of 2, and the second and last pattern 1 (LSB)has a stripe width of 2. A Gray code of 2 bits can encode 4(=2²) uniquestripes. In a typical implementation, each dimension of the frame bufferis encoded in the same manner using the same Gray code of 2 bits. Toencode the x dimension, each row of the image will contain the samepattern, which creates a set of vertical stripes. To encode the ydimension, each column of the image will contain the same pattern,creating horizontal stripes. FIG. 8B shows a set of detail patternsgenerated as described above for x and y dimensions. Patterns (ax0p) and(ax1p) correspond to the x dimension, and patterns (ay0p) and (ay1p)correspond to the y dimension. Patterns (ax0n) and (ax1n) are negativesof patterns (ax0p) and (ax1p), and patterns (ay0n) and (ay1n) arenegatives of patterns (ay0p) and (ay1p). To encode 1024 pixels along onedimension using the detail patterns of FIG. 8B, 256 repetitions of thedetail patterns will be used along the dimension. However, only the twodetail patterns will be needed to encode 1024 pixels, compared to thenine patterns needed with regular Gray coding for the same resolution.

The number of repetitions of the detail patterns needed to encode eachdimension of the frame buffer may be reduced by mapping one Gray code toa group of pixels—this will also increase the size of the features andsimplify detection. For example, if 16 pixels are encoded with 2 bits(two patterns), there will be a 1:4 correspondence between Gray code andpixels, for a minimum stripe width of 8 pixels, as shown in FIG. 8C. Theimages will be offset by 4 pixels (compare pattern 0 to pattern 1).Thus, there will be data for each 4 pixels, but the smallest feature tobe identified is 8 pixels. To encode 1024 pixels along one dimensionusing the Gray code patterns in FIG. 8C, 64 repetitions of the detailpatterns will be used along the dimension, compared to the 256repetitions needed when there is a 1:1 correspondence between Gray codeand pixels.

When decoding the captured image of test patterns, the decoding systemneeds to decide on a brightness threshold to use in deciding whethereach pixel in the captured image should be classified as “on” or “off.”One way to simplify this process is to project both a positive andnegative image for each test pattern. The negative images are theinverses of the positive images (black stripes are made white, and viceversa). Then, pixels in the positive and negative images are compared.Pixels that are brighter in the positive image are classified as “on”and those that are dimmer are classified as “off.” The final image willbe a differential image. This greatly simplifies decoding, although itdoubles the number of patterns captured. FIGS. 7B and 8B show positiveand negative test patterns.

The example above results in two global patterns (positive and negativepatterns) and eight detailed patterns (positive and negative patterns).These patterns can be used to encode any size of frame buffer. Thedetail patterns will be repeated along each respective dimension tocover the desired size of frame buffer.

In the example above, the detail patterns are generated using 2 bits peraxis or dimension. In another implementation, the detail patterns couldbe generated using 3 bits per axis or dimension, which would allowtracking of larger distortions, improve error rejection, or somecombination of the two during decoding. In theory, the detail patternscould also be generated using 1 bit per axis or even 1 bit per pixel,but this would reduce the precision or accuracy of the decoding comparedto 2 bits per axis.

The sequence of patterns is projected by the projector and captured bythe camera, producing an image of each projected pattern in the cameraspace. These images are then decoded to determine a mapping between theprojector space and the camera space. In one implementation, to generatea mapping between the projector space and the camera space, the capturedimage of the global pattern (captured global image) is first decoded.For example, image recognition may be used to find the known features inthe captured global image and to decide which feature in the capturedglobal image corresponds to which feature in the frame buffer globalpattern. This process will result in a coarse correlation between theprojector space and the camera space. For the example where the globalpattern uses four features with known positions in the frame buffer,this process will generate a map with four points in the projector space(or frame buffer) and corresponding four points in the camera space.

Next, the captured images of the detail patterns (captured detailedimages) are decoded to assign each camera pixel an x, y value thatcorresponds to the Gray code of the pixel position. Pixels may also beassigned a preliminary confidence value based on the local brightnesscontrast and/or other image quality factors. This is a standard Graycode decoding process, with the exception that the result will consistof repeating “tiles” from [0, M), where M is the number of distinct Graycode values used per axis. M=2^(m), where m is the number of bitsencoding the axis. In the example where each axis is encoded with a 2bit Gray code, M=4. M may also be referred to as low value count.

Each pixel in the frame buffer corresponds to a region, normallyconsisting of multiple adjacent pixels, in the captured image. The twolow bits corresponding to each pixel are given by the Gray code decodingprocess. Now, the high bits are needed to obtain the actual address ofthe pixel in the frame buffer. Using the decoded global pattern data, amathematical model, called a bootstrap model herein, is generatedbetween the frame buffer space and the camera space. In oneimplementation, this may be a simple linear model of the form y=A+BX+CY;x=D+EX+FY, where x, y represent points in the frame buffer space and X,Y represent points in the camera space. Next, a bootstrap region isselected and decoded. The selected bootstrap region should be one forwhich the bootstrap model is a good approximation. In oneimplementation, the selected bootstrap region is the region bounded bythe four known points in the global image. In an example where thepoints are spaced apart by 32 pixels (i.e., D in FIG. 7A is 32 pixels)and the stripes of the detail pattern are 8 pixels wide, the regionbounded by the four points will have two repeats of the detail patternin each dimension. Decoding the bootstrap region includes inferring thehigh bits for each pixel in the bootstrap region.

One example of inferring the high bits for each pixel includescalculating the expected frame buffer value using the bootstrap model.Next, the nearest frame buffer value to the expected value that has thesame low bits as the captured image (of x and y) is calculated. If thedistance to nearest frame buffer value is no more than a distancethreshold, the pixel is assigned that value; otherwise, the pixel is notdecoded. In one example, the distance threshold is 0.25×M, where M isthe number of distinct Gray code values used per axis or the low valuecount. For the case where each axis is encoded by a 2 bit Gray code,M=4. A different distance threshold may be used. Further, the distancemetric may be applied either to each axis independently or as a combineddistance.

To illustrate the process of inferring high bits, consider an examplewith 2 low bits encoded with Gray code. The low bits may have a value of0, 1, 2, or 3. For a pixel which the bootstrap model predicts an x valueof 100.1, the reconstructed value depends on the decoded value asfollows:

Case 0: The nearest integer value to 100.1 such that result % 4=0 is100. This has a distance of 0.1, which is no more than a distancethreshold of 1. Thus the pixel is accepted with a result of 100.

Case 1: The nearest integer value to 100.1 such that result % 4=1 is101. This has a distance of 0.9, which is no more than a distancethreshold of 1. Thus the pixel is accepted with a result of 101.

Case 2: The nearest integer value to 100.1 such that result % 4=2 is102. This has a distance of 1.9, which is more than a distance thresholdof 1. Thus the pixel is rejected.

Case 3: The nearest integer value to 100.1 such that result % 4=3 is 99.This has a distance of 1.1, which is no more than a distance thresholdof 1. Thus the pixel is rejected.

Next, a region that substantially overlaps the previously decoded areaand contains a new region that has not been decoded is selected. A newmathematical model is built based on the previously decoded data. Then,the new area is decoded using the same technique used to decode thebootstrap region. This is repeated until the entire image is decoded.The result will be a detailed map with points in the frame buffer andcorresponding points in the camera space.

FIG. 9 illustrates a process for calibrating a WHUD to produce virtualimages that are overlapping and aligned in an eye space. At 300, acamera (e.g., 200 in FIG. 2) is placed relative to the WHUD (e.g., 100in FIG. 2) at approximately the same position as an eye would be if adisplay UI is to be presented to the eye by the WHUD. In this position,the camera can receive or detect light signals from the WHUD (or fromthe transparent combiner of the WHUD).

At 304, for each virtual image V(i,j), where i represents a particularcolor channel and j represents a particular exit pupil, each testpattern from the set of test patterns is projected to a field of view ofthe camera in a defined sequence. The test patterns may be the reducedimage coding patterns (e.g., the global and detailed Gray code patternsin FIGS. 7B and 8B) or any other test patterns that enable mapping ofthe projector space to the camera space. FIGS. 10A-1, 10A-2, 10A-3,10A-4, and 10A-5 show screenshots of actual test patterns used in acalibration process according to one example. Only the positive testpatterns are shown in FIGS. 10A1-10A-5, but the full set of testpatterns also includes the inverses of the patterns shown in FIGS.10A1-10A-5. The same set of test patterns is projected for all virtualimages, with the exception that the test patterns are cropped toapproximately the correct display region to minimize stray light. Also,when projecting the patterns through a particular color channel, whiteis changed to the color of that channel. Thus, for example, red andblack stripes will be projected if the color channel of interest is red.

Displaying each test pattern in a field of view of the camera includesrendering the test pattern into the projector frame buffer, generatinglight signals representative of the frame buffer data, and projectingthe light signals to the transparent combiner (114 in FIG. 2) in thefield of view of the camera (200 in FIG. 2). Projection of the lightsignals to the transparent combiner may involve an optical splitter (116in FIG. 2) or other exit pupil replication structure as describedpreviously, e.g., if the transparent combiner is a holographic combiner.For each test pattern displayed in the field of view of the camera (orprojected to an exit pupil of the display via the transparent combiner),the camera captures an image of the displayed test pattern. FIGS. 10B-1,10B-2, 10B-3, 10B-4, and 10B-5 show screenshots of actual capturedimages corresponding to the test patterns in FIGS. 10A-1, 10A-2, 10A-3,10A-4, and 10A-5, respectively.

Returning to FIG. 9, in one implementation of 304, the sequence of testpatterns is projected separately for each unique combination of colorchannel and exit pupil. That is, given a unique combination of colorchannel and exit pupil, the sequence of test patterns is projected.Then, this is repeated for each of the remaining unique combinations ofcolor channel and exit pupil.

In another implementation of 304, multiple copies of each test patternin a sequence of test patterns are generated. For each position in thesequence, the multiple copies of the test pattern for that position areprojected simultaneously, each copy following an optical path defined bya unique combination of color channel and exit pupil. For example, given3 color channels and exit pupil EP0 for which test patterns are to beprojected simultaneously, the process may be as follows: for eachposition in the sequence of test patterns, a copy of the test patternfor that position for V(i,j) is generated, where i in 0 . . . 2 (e.g.,0=red channel, 1=green channel, 2=blue channel) and j=0 (e.g., 0=exitpupil EPO), and the copies are projected simultaneously—this may berepeated for other exit pupils. In the implementation where multiplecopies of a test pattern are projected for each position in thesequence, the camera may capture the multiple projected copies of thetest pattern, which means that the captured image will contain data formore than one unique combination of color channel and exit pupil. If themultiple projected copies are distinguished by color, the camera willneed to be able to distinguish between colors to allow the informationfor each unique combination of color channel and exit pupil to beisolated.

At 308, for each virtual image V(i,j), the captured images are decodedto find correspondence between projector frame buffer pixels and camerapixels. This results in a mapping P(i,j)->C(i,j) from the projectorspace to the camera space. In one example, the mapping P(i,j)->C(i,j)may be explicit correlations between the projector space and cameraspace for every projector frame buffer pixel, which may be in the formof a lookup table. (If the test patterns are generated using the reducedimaging coding previously described, then the companion decoding processpreviously described may be used to map between the projector space andcamera space.) However, other models are possible, such as 2D polynomialmodels, splines, light-path models (i.e., models that more accuratelyreflect the actual optics of the system), or other abstract models. Theother models may be obtained by performing at least one transformationbetween the points in the projector space and the points in the cameraspace, such as a linear transformation, a non-linear optimization, anaffine transformation, a neural network-based transformation, and thelike.

At 312, for each mapping P(i,j)->C(i,j) from the projector space to thecamera space corresponding to each virtual image V(i,j), a mappingC(i,j)->E from the camera space to the eye space is acquired. Themapping C(i,j)->E from the camera space to the eye space may be anexisting mapping or may be determined at the beginning of thecalibration process. In one example, the C(i,j)->E mapping may bederived by theoretical modeling of the optics of the camera (includingany lens) and the physical placement of the camera relative to thetransparent combiner. In another example, the C(i,j)->E mapping may bederived or refined by placing a physical reference object, e.g., a flatsurface with a checkerboard, at the same position relative to the camerathat the virtual image from the transparent combiner will appear andmaking measurements based on this setup. In this case, the model betweenthe camera and the image of the reference object would match the modelbetween the camera and the desired virtual image. Developing a mappingbetween the camera and a flat reference surface is a common problem withmultiple well documented solutions. See, for example, a tutorial oncamera calibration in the documentation for the popular OpenCV machinevision library (“Camera Calibration with OpenCV, Open CV 2.4.13.7Documentation” Web. 9 Aug. 2018).

One or more regions of interest are identified for each virtual imageV(i,j) in the eye space. The regions of interest may include regions ofthe virtual image where the virtual image overlaps all other virtualimages in the eye space. The regions of interest may further includeregions of the virtual image where the virtual image does not overlapany other virtual image or overlaps only some of the other virtualimages. The regions of interest may exclude regions of the virtual imagethat are not needed or usable to form a display UI that is seen by auser. The regions of interest may also be optimized for other objectivesincluding size, position, or image quality.

Within each defined region of interest, for each virtual image V(i,j), amapping E->P(i,j) from the eye space to the projector space isdetermined using the mapping P(i,j)->C(i,j) from the projector space tothe camera space and the mapping C(i,j)->E from the camera space to theeye space. This may either be a direct mathematical transform or mayinclude additional filtering (for example, noise filtering,interpolation, or extrapolation). The eye space E is common across allvirtual images. Therefore, for any point in the eye space E within theoverlapping region of interest, applying the appropriate mappingE->P(i,j) generates a coordinate in the corresponding projector space.

At 316, the E->P(i,j) mapping for each defined region of interest foreach virtual image V(i,j) may be stored in the WHUD, e.g., in memory 136or storage 138 in FIG. 3, and used for subsequent generation of acorresponding distorted source image S′(i,j). If the display has N 1separately addressable exit pupils and K≥1 color channels, there willgenerally be N times K virtual images and corresponding N times K E->P(eye space to projector space) mappings. Alignment of virtual images inthe target eye space becomes relevant when N times K>1, i.e., there willbe two or more virtual images in the target eye space.

The acts above have generated for each virtual image V(i,j), a mappingE->P(i,j) from common eye space E to unique projector space P(i,j) forthat virtual image within the defined region(s) of interest. Now, adisplay UI may be displayed within which exhibits both geometriccorrection and alignment between virtual images. To present a display UIin the eye space, distorted source images are generated. This may alsobe regarded as rendering the source images into the eye space. Togenerate the distorted source image, for each pixel of the distortedsource image, the corresponding point in the eye space is found usingthe mapping E->P(i,j). The lookup may apply a simple “nearest neighbor”,bi-linear interpolation, or other common sampling algorithm. Thesampling may also explicitly consider the area of each pixel, mappingthe points at the corners or bound of the pixel. In one example, thesampling algorithm may perform the interpolation in a linear colorspace. When the frame buffer pixels are projected to the eye space, thedistorted source images will form virtual images in the eye space thatare aligned at least within the defined regions of interest. The sourceimages can be distorted when rendering the source images into the framebuffer or after rendering the source images into the frame buffer. Inthe latter case, the display controller (e.g., 142 in FIG. 3) maygenerate the distorted source images. Projection of the distorted sourceimages includes generating light and modulating the light with thedistorted source images. In one example, modulating the light with thedistorted source images may include the display controller providing thedistorted source image data to the light engine driver (146 in FIG. 3),which then generates controls for the light engine (118 in FIG. 3)according to the distorted source image data.

Changes in temperature of the projector (and other elements in the lightprojection path) may affect the nonlinear distortion experienced by eachvirtual image formed in the eye space. To account for this, acts 304-316may be performed at a first temperature T1 of the projector—anyreasonable element in the projector, such as a body of the light engineor cavity in the projector, may provide an indication of the temperatureof the projector. Acts 304-316 may be repeated for at least anothersecond temperature T2 of the projector that is different from the firsttemperature T1. This will yield a mapping E->P(i,j)_(T1) at the firsttemperature T1 and a mapping E->P(i,j)_(T2) at the second temperature.In this case, distorting the source images may include: for each framebuffer pixel in each projector space P(i,j) that generates virtual imageV(i,j), look up a corresponding point at T1 in the eye space using thecorresponding mapping E->P(i,j)_(T1), look up a corresponding point atT2 in the eye space using the corresponding mapping E->P(i,j)_(T2), andinterpolate between these corresponding points, or extrapolate fromthese corresponding points, based on the current temperature of theprojector to obtain the corresponding point in the eye space for thecurrent temperature of the projector. The above requires measuring thetemperature of the projector, and the temperature measurements may beprovided to the application processor. This technique additionally oralternatively may be applied to other elements of the light path usingother temperature measurement points, e.g., the temperature of thetransparent combiner may be relevant and may not be well correlated withthe temperature of the projector. In the case where the temperature ofthe transparent combiner may be more relevant, the distortion models(E->P(i,j)) can be determined for different temperatures of thetransparent combiner and used for subsequent generation of the distortedsource images.

FIG. 11 shows an image of a logo (display UI) that is to be displayed toa user according to one example. FIG. 12 shows a screenshot of thedistorted source images in the projector frame buffer. This example has4 frame buffer regions from which distorted source images are to beprojected to 4 spatially-separated exit pupils. Each frame buffer regionmay contain a distorted source image for each of the color channel usedby the projector, which means that there may be more than one distortedsource image in each frame buffer region. When these distorted sourceimages are projected to the eye space, they will form a display UI thatappears as the original image of the logo to be displayed.

Although the calibration process has been described with reference to aWHUD having a scanning laser projector, the calibration process need notbe limited as such. For example, the calibration process may be appliedto a WHUD that uses digital light projector (DLP) or organiclight-emitting (OLED) projector.

The foregoing detailed description has set forth various implementationsor embodiments of the devices and/or processes via the use of blockdiagrams, schematics, and examples. Insofar as such block diagrams,schematics, and examples contain one or more functions and/oroperations, it will be understood by those skilled in the art that eachfunction and/or operation within such block diagrams, flowcharts, orexamples can be implemented, individually and/or collectively, by a widerange of hardware, software, firmware, or virtually any combinationthereof. In one implementation or embodiment, the present subject mattermay be implemented via Application Specific Integrated Circuits (ASICs).However, those skilled in the art will recognize that theimplementations or embodiments disclosed herein, in whole or in part,can be equivalently implemented in standard integrated circuits, as oneor more computer programs executed by one or more computers (e.g., asone or more programs running on one or more computer systems), as one ormore programs executed by on one or more controllers (e.g.,microcontrollers) as one or more programs executed by one or moreprocessors (e.g., microprocessors, central processing units, graphicalprocessing units), as firmware, or as virtually any combination thereof,and that designing the circuitry and/or writing the code for thesoftware and or firmware would be well within the skill of one ofordinary skill in the art in light of the teachings of this disclosure.

When logic is implemented as software and stored in memory, logic orinformation can be stored on any processor-readable medium for use by orin connection with any processor-related system or method. In thecontext of this disclosure, a memory is a processor-readable medium thatis an electronic, magnetic, optical, or other physical device or meansthat contains or stores a computer and/or processor program. Logicand/or the information can be embodied in any processor-readable mediumfor use by or in connection with an instruction execution system,apparatus, or device, such as a computer-based system,processor-containing system, or other system that can fetch theinstructions from the instruction execution system, apparatus, or deviceand execute the instructions associated with logic and/or information.

In the context of this disclosure, a “non-transitory processor-readablemedium” or “non-transitory computer-readable memory” can be any elementthat can store the program associated with logic and/or information foruse by or in connection with the instruction execution system,apparatus, and/or device. The processor-readable medium can be, forexample, but is not limited to, an electronic, magnetic, optical,electromagnetic, infrared, or semiconductor system, apparatus or device.More specific examples of the processor-readable medium are a portablecomputer diskette (magnetic, compact flash card, secure digital, or thelike), a random-access memory (RAM), a read-only memory (ROM), anerasable programmable read-only memory (EPROM, EEPROM, or Flash memory),a portable compact disc read-only memory (CDROM), digital tape, andother non-transitory medium.

The above description of illustrated embodiments, including what isdescribed in the Abstract of the disclosure, is not intended to beexhaustive or to limit the embodiments to the precise forms disclosed.Although specific embodiments and examples are described herein forillustrative purposes, various equivalent modifications can be madewithout departing from the spirit and scope of the disclosure, as willbe recognized by those skilled in the relevant art. The teachingsprovided herein of the various embodiments can be applied to otherportable and/or wearable electronic devices, not necessarily theexemplary wearable electronic devices generally described above.

1. A method of calibrating a wearable heads-up display, the methodcomprising: for each virtual image to be formed by the wearable heads-updisplay in a target eye space, determining a distortion model for use indistorting a source image corresponding to the virtual image, whereindetermining the distortion model comprises: projecting a set of testpatterns by a projector of the wearable heads-up display in a selectsequence; capturing an image of each test pattern projected by theprojector by a camera; determining a first mapping that maps points in aprojector space to points in a camera space from the captured images;acquiring a second mapping that maps points in the camera space topoints in the target eye space; determining a third mapping that mapspoints in the target eye space to points in the projector space based onthe first mapping and the second mapping; and storing the third mappingas the distortion model for use in distorting the source imagecorresponding to the virtual image.
 2. The method of claim 1, whereindetermining a first mapping that maps points in a projector space topoints in a camera space from the captured images comprises decoding thecaptured images to find correspondence between projector frame bufferpixels and camera pixels.
 3. The method of claim 2, wherein decoding thecaptured images to find correspondence between projector frame bufferpixels and camera pixels comprises detecting Gray codes in the capturedimages.
 4. The method of claim 1, wherein determining a third mappingthat maps points in the target eye space to points in the projectionspace based on the first mapping and the second mapping comprisesidentifying at least one region of interest in the target eye space anddetermining the third mapping for the at least one region of interest inthe target eye space.
 5. The method of claim 1, wherein determining thedistortion model for use in distorting a source image corresponding tothe virtual image is performed at a first temperature to obtain adistortion model at the first temperature and at a second temperature toobtain the distortion model at the second temperature.
 6. The method ofclaim 1, wherein projecting a set of test patterns by the projector ofthe wearable heads-up display in a select sequence comprises: for eachposition in the select sequence, generating copies of the test patternat the position and projecting the copies of the test pattern by theprojector of the wearable heads-up display, wherein each of the copiescorresponds to a unique combination of exit pupil and color channel ofthe wearable heads-up display.
 7. The method of claim 6, whereincapturing an image of each test pattern projected by the projector by acamera comprises capturing an image having image portions correspondingto the copies of the test patterns by the camera.
 8. The method of claim7, wherein determining a first mapping that maps points in a projectorspace to points in a camera space from the captured images comprisesdetermining the first mapping for each unique combination of colorchannel and exit pupil using the image portions of the captured imagecorresponding to the unique combination of color channel and exit pupil.9. The method of claim 1, further comprising generating the set of testpatterns prior to determining a distortion model for use in distorting asource image corresponding to the virtual image, wherein generating theset of test patterns comprises generating at least one pattern carryingcodes.
 10. The method of claim 9, wherein generating at least onepattern carrying codes comprises generating the at least one patternwith a Gray code of m bits, where m
 1. 11. The method of claim 9,wherein determining a first mapping that maps points in a projectorspace to points in a camera space from the captured images comprisesdecoding the at least one pattern carrying codes.
 12. The method ofclaim 9, wherein generating the set of test patterns further comprisesgenerating at least one additional pattern having features with knownpositions in a frame buffer of the projector.
 13. The method of claim12, wherein determining a first mapping that maps points in a projectorspace to points in a camera space from the captured images comprisesdecoding the at least one pattern carrying codes and the at least oneadditional pattern having features with known positions in the framebuffer of the projector.
 14. The method of claim 1, wherein projecting aset of test patterns by a projector of the wearable heads-up display ina select sequence comprises transmitting the set of test patterns to aprocessor of the wearable heads-up display, wherein the processor of thewearable heads-up display renders the set of test patterns into a framebuffer of the projector in the select sequence.
 15. The method of claim1, wherein storing the third mapping as the distortion model for use indistorting the source image corresponding to the virtual image comprisesstoring the third mapping in a non-transitory processor-readable storagemedium of the wearable heads-up display.
 16. The method of claim 1,wherein capturing an image of each test pattern projected by theprojector by a camera comprises positioning the camera relative to thewearable heads-up display to capture at least a portion of imagesprojected by the wearable heads-up display.
 17. A wearable heads-updisplay calibration system, comprising: a wearable heads-up displaycomprising a projector; a camera positioned and oriented to captureimages projected by the projector; a calibration processorcommunicatively coupled to the wearable heads-up display and camera; anda non-transitory processor-readable storage medium communicativelycoupled to the calibration processor, wherein the non-transitoryprocessor-readable storage medium stores data and/orprocessor-executable instructions that, when executed by the calibrationprocessor, cause the calibration processor to determine a distortionmodel that maps points from a target eye space to points in a projectorspace for each virtual image to be produced in the target eye space bythe wearable heads-up display.
 18. The wearable heads-up display ofclaim 17, wherein the wearable heads-up display further comprises aprocessor, and wherein the calibration processor is communicativelycoupled to the processor of the wearable heads-up display.
 19. A methodof forming a display UI on a wearable heads-up display, comprising:generating N times K distorted source images, wherein N≥1 and K≥1 and Ntimes K>1, by applying N times K different distortion models to K sourceimages, wherein each of the K different distortion models maps pointsfrom a target eye space to points in a projector space corresponding toone of the K source images, wherein N is the number of separatelyaddressable exit pupils of the display and K is the number of colorchannels of the display; and projecting at least two of the distortedsource images from the projector space to the target eye space, whereinthe at least two of the distorted source images form at least twovirtual images in the target eye space that are aligned in the targeteye space.
 20. The method of claim 19, wherein generating N times Kdistorted source images comprises applying the N times K differentdistortion models to the K source images when rendering the K sourceimages into a projector frame buffer of the wearable heads-up display.21. The method of claim 19, wherein generating N times K distortedsource images comprises applying the N times K different distortionmodels to the K source images after rendering the K source images into aprojector frame buffer of the wearable heads-up display.
 22. The methodof claim 19, wherein projecting at least two of the distorted sourceimages to the target eye space comprises generating light and modulatingthe light with the at least two of the distorted source images.